Repository navigation
Opt-in first-recorded-failure ordering for all() with a per-run marker (#307) - #348
Merged
Merged
Conversation
When several members of an all() group had failed by the time a task reached it and every member had settled, the worker raised the failure with the lowest input position. Workflow's embedded engine raises the failure recorded first, so the exception a workflow caught depended on whether its task ran before or after the other members finished. A task reaching a settled group now raises its first recorded failure. It records the member it raised with the #338 decision side effect only when that member is not also the lowest-position failure, so its commands are unchanged whenever both rules agree. Replay raises the member a decision names. A settled group with no decision after which the run closed or did further durable work came from a released worker and keeps the position rule and its clock behavior. Native Server fixtures recorded by published 2.19.1 (position rule, complete and in flight) and by this version (decision, deciding task) cover both rules. The selection-gap fixture's frontier expectation is retired as same-protocol evidence under a corrected fixture. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This was referenced Nov 19, 2026
Review of the previous head showed that deciding from history alone is unsound. A released worker could catch a settled group's lowest-position failure and finish the task with no command, for example while awaiting a select() handle. A later task then saw no decision and nothing after the group, took it for a new frontier and chose another member. An update answered in between had already reported the first choice. Each run now records the version marker durable_workflow.parallel_failure_order once, directly before the first all-group that contains an activity or child workflow, when that group is first scheduled. A marked run raises the first failure in durable event order, whether or not the other members have settled, and needs no per-group decision because history is append-only. A run whose first such group was recorded without the marker keeps the released rule (position when settled, the #338 decision when raised early) for its whole life, on every task, query and update. A run with a pending cancellation request at that point records no marker. Native and PHP Server built from main accept the marker. Published 2.19.1 refuses a marked run as nondeterministic rather than choosing differently. Fixtures recorded by 2.19.1 (zero-command catch, update before and between this version's tasks, select() cancellation, settled group in flight) replay under the released rule. Fixtures recorded by this version replay under the new rule. The selection-gap fixture is kept as released behaviour. Three fixtures of a new run's first group emission are superseded because they now include the marker. Docs now say Python follows the contract's event order where embedded Workflow compares millisecond timestamps, then index (durable-workflow/.github#415). The select() base-sequence bug found in review is #350. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Published 2.19.1 cannot replay a run that recorded the parallel failure-order marker. The Server marks the workflow task replay-blocked with no retry, and only a repair resumes it, which can hand it to an old worker again. Rolling back would do the same to every marked run. Follow the two-phase pattern of workflow#1239: - Every worker reads and follows a recorded marker, on tasks, queries and updates. - Writing it is off by default. Worker(record_parallel_failure_order=True) or DURABLE_WORKFLOW_RECORD_PARALLEL_FAILURE_ORDER=true turns it on. Any other environment value is refused when the Worker is created. replay() and Replayer.replay() take the same keyword. - With the default, new runs send exactly the commands 2.19.1 sends. - The next minor release flips the default (#351). This removes the same-protocol supersession of new-run emission fixtures: parallel-group-metadata-emission and the two completion-limit fixtures are unchanged and pass again. New fixtures cover the first task with recording on and off. The marked native and PHP histories replay under the new rule without the setting. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Move the failure-order entry to Unreleased. Name 2.20.0 as the last release that cannot replay a marked run. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Related: #307, which stays open for the explicit
propagate,cancel_remainingandsettlepolicies (durable-workflow/.github#232). Follows #338/#341. The default flip is tracked in #351.Semantics established
finishParallelDispatchraises a recorded group failure ahead of open members, and nothing is cancelled.ParallelFailureSelectorpicks the earliestrecorded_atin milliseconds, then the lower leaf index.selection_resume): a failed member wakes the parent. A success wakes it only once the whole group has succeeded. Nothing is cancelled..githubconformance/parallel-groups, lines 103 and 121): first failure in durable event order.Change: per-run marker, two-phase rollout (pattern of workflow#1239)
durable_workflow.parallel_failure_order(v1), directly before its first all-group containing an activity or child, raises the first failure in durable event order, settled or not. It records no per-group decision, because history is append-only. Every worker of this release follows a recorded marker on tasks, queries and updates, whatever its setting.Worker(record_parallel_failure_order=True)orDURABLE_WORKFLOW_RECORD_PARALLEL_FAILURE_ORDER=trueturns writing on. Any other value is refused when theWorkeris created.replay()andReplayer.replay()take the same keyword. Default off: new runs send byte-identical commands to 2.20.0 (and 2.19.1).Live (Server built from main)
py307.fail#1. 2.19.1 also replays that history.py307.fail#3. 2.19.1 refuses that history.await_resulton a select handle)fail-m0throughoutfail-m0, compensationfail-m0fail-m0/fail-m0fail-m1, updatefail-m1Fixtures (no existing fixture changed or retired)
failure-order-opt-in-first-task-records-marker(recording on);failure-order-default-first-task-sends-released-commands(recording off);parallel-failure-order-opt-in-new-run-emission.parallel-group-metadata-emission, the two completion-limit fixtures andbusiness-clock-parallel-failure-selection-gapare unchanged and pass.Tests
tests/test_parallel_failure_order.py:Related
_ordinary_all_groupread the recorded policy. That check only matters with the draft explicit-propagateoption, which is disabled.Checks
🤖 Generated with Claude Code